EDA Final Report

Introduction

In this section, explain why you chose this topic, and the questions you are interested in studying. Include a brief description of how you found the data, and clear instructions on where the reader can find the data.

International travel has become so easy that anyone can just grab his/her bags and documents and starts a new journey. As one of the most tourists importing and exporting countries, the USA has always played a significant role in world’s tourism industry. Therefore, we want to study people who traveled to and from the USA by investigating the Travel and Tourism Statistical data provided by The National Travel & Tourism Office (NTTO). To be more precise, we want to answer following questions:

• What are the most popular international travel destinations for Americans?

• What are the main regions of people who travel to the USA?

• What are the purposes and spending of travels to and from the USA?

• Does the economy matter for the answers to above questions?

Data is collected from two resources,

• National Travel & Tourism Office (NTTO):
  • Inbound Information
  • Outbound Information
• World Bank:

http://tinet.ita.doc.gov/outreachpages/inbound.general_information.inbound_overview.asp

http://tinet.ita.doc.gov/view/m-2016-I-001/index.asp

http://travel.trade.gov/research/monthly/arrivals/index.asp

NTTO monthly spending NTTO yearly spending NTTO monthly inbound NTTO monthly outbound Word Bank GDP

From NTTO, … NTTO do not provide raw data

Although NTTO provides plenty of information we need, it is not easy to collect and combine them together. There is much unavoidable manual work to do in the collecting process because of the inconsistent formatting (data formats may vary from year to year), file types (some are pdfs) and location of these files. For example, in order to get monthly arrival information per region, we have to download each year’s excel files, and then clean, select and combine them into a final master table that is easier to manipulate with in R.

In the end, we collect several master tables from NTTO: yearly spending per region with three categories, total monthly spending with three categories, monthly inbound visitation per country and monthly outbound visitation per country. We may use other tables as complementary tables.

From World Bank,… On this website, we are trying to get GDP per country per year, as a measurement of economy.

GDP data from World Bank is easier to clean. Further, we will load these tables to R and use dplyr and tidyr to fit the data for our analysis.

Team

List team members and a description of how each contributed to the project. (If you’re working alone, briefly describe the stages of the project.)

• Lucy Lu: finding and organizing the data, compiling visualization and stories, creating the interactive app, focusing on the import part.

• Hanlin Wu: cleaning the data, testing various visualizations, creating the interactive app, focusing on the export part.

Analysis of Data Quality

Provide a detailed, well-organized description of data quality, including textual description, graphs, and code.

library(tidyverse)
library(viridis)
library(ggvis)
library(extracat)
library(plotly)
library(zoo)

raw_ntto_spend_y <- tbl_df(read.csv("clean_data/clean_yearly_spending_region.csv", header = TRUE))
# raw_ntto_spend_m <- tbl_df(read.csv("clean_data/Monthly_Exports_Imports_Balance.csv", header = TRUE))
raw_ntto_inbound_m <- tbl_df(read.csv("clean_data/clean_monthly_visitation_inbound_country.csv", header = TRUE))
raw_ntto_outbound_m <- tbl_df(read.csv("clean_data/clean_monthly_us_to_international.csv", header = TRUE))
raw_wb_gdp <- tbl_df(read.csv("clean_data/API_NY.GDP.MKTP.CD_DS2_en_csv_v2.csv", header = TRUE, skip=4))

Data we have are: monthly spending from NTTO, yearly spending from NTTO, monthly inbound from NTTO, monthly outbound from NTTO and GDP from Word Bank. In order to analyze data quality of our data, we need first to take a look at them.

Output of str has been truncated to get a more compact look in report

• Yearly Spending from NTTO

str(raw_ntto_spend_y, list.len=5)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   48 obs. of  13 variables:
 $ region  : Factor w/ 8 levels "Africa","Asia-Pacific",..: 1 1 1 1 1 1 2 2 2 2 ...
 $ type    : Factor w/ 2 levels "Payments (imports)",..: 2 2 2 1 1 1 2 2 2 1 ...
 $ category: Factor w/ 3 levels "Education","Medical/Short-Term Workers",..: 1 2 3 1 2 3 1 2 3 1 ...
 $ X2006   : int  930 208 1390 137 29 1740 8566 836 23160 626 ...
 $ X2007   : int  961 183 1594 161 30 2435 9455 810 24756 667 ...
  [list output truncated]

• Monthly Spending from NTTO

# str(raw_ntto_spend_m, list.len=4)

• Monthly Inbound from NTTO

str(raw_ntto_inbound_m, list.len=5)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   132 obs. of  32 variables:
 $ year                       : int  2015 2015 2015 2015 2015 2015 2015 2015 2015 2015 ...
 $ MONTH                      : Factor w/ 12 levels "APRIL","AUGUST",..: 5 4 8 1 9 7 6 2 12 11 ...
 $ TOTAL.ARRIVALS             : Factor w/ 132 levels "2,592,882","2,624,018",..: 91 70 109 119 124 110 130 131 120 121 ...
 $ CANADA                     : Factor w/ 132 levels "1,007,315","1,029,277",..: 63 39 109 96 89 50 112 120 71 44 ...
 $ MEXICO                     : Factor w/ 132 levels "1,004,126","1,007,303",..: 44 18 27 50 52 38 53 57 35 55 ...
  [list output truncated]

• Monthly Outbound from NTTO

str(raw_ntto_outbound_m, list.len=4)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   102 obs. of  14 variables:
 $ Year   : int  2015 2015 2015 2015 2015 2015 2015 2015 2015 2015 ...
 $ Regions: Factor w/ 17 levels "Africa","Asia",..: 8 6 2 16 7 15 13 1 17 12 ...
 $ Jan    : Factor w/ 102 levels "1,707,635","1,716,026",..: 96 91 59 16 47 89 20 49 31 37 ...
 $ Feb    : Factor w/ 102 levels "1,514,915","1,580,058",..: 79 94 59 16 50 98 17 39 30 35 ...
  [list output truncated]

• GDP from Word Bank

str(raw_wb_gdp, list.len=6)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   264 obs. of  62 variables:
 $ Country.Name  : Factor w/ 264 levels "Afghanistan",..: 11 1 6 2 5 8 250 9 10 4 ...
 $ Country.Code  : Factor w/ 264 levels "ABW","AFG","AGO",..: 1 2 3 4 5 6 7 8 9 10 ...
 $ Indicator.Name: Factor w/ 1 level "GDP (current US$)": 1 1 1 1 1 1 1 1 1 1 ...
 $ Indicator.Code: Factor w/ 1 level "NY.GDP.MKTP.CD": 1 1 1 1 1 1 1 1 1 1 ...
 $ X1960         : num  NA 5.38e+08 NA NA NA ...
 $ X1961         : num  NA 5.49e+08 NA NA NA ...
  [list output truncated]

The internal structure of these data indicates several data quality issues:

• Not tidy:
  • It is better in our case to have a columns named Year instead of each year has its own column.
• Messy column names:
  • Some columns like X1960 is weird and we should remove the extra “X”.
• Wrong types of values:
  • Some numerical value have been recognized as string (e.g. “1,707,635”, “$1,372”)
• Inconsistency of categorical variable values and column names:
  • category in Yearly Spending and X.1 (The content of this column represents spending category as well) in Monthly Spending have different number of factors and for the same category. For example, we suspisious that “Education related” and “Medical/Education/Workers Spending” should refer to the same category.
  • In some table month is represented as first three letters (e.g. Jan) and in other table is represented as full name (e.g. APRIL).
  • In consistency of upper/lower case of strings (e.g. Year and year)
• Missing values
  • In order to show missing value more clearly, we pre-processed data, the detail about how to clean data will be discussed later in Main Analysis part.
  • As what are shown in the graph below, we only have some missing values in GDP data for countries we are interested. But the missing values are either before 1990 nor after 2016, which would not influence our analysis.

country_str <- 'Argentina|Australia|Brazil|Canada|Chile|China|Colombia|France|Germany|India|Italy|Japan|Mexico|Netherlands|Russia|Saudi Arabia|South Korea|Singapore|Spain|Sweden|Switzerland|Taiwan|United Kingdom|United States' 
tidy_ntto_spend_y <- raw_ntto_spend_y %>%
   gather(x_year, spend, -region, -type, -category) %>% 
   separate(x_year, c("X", "Year"), sep="X") %>% 
   select(-X) %>% 
   select(Region=region, Type=type, Category=category, Year=Year,Spend=spend) %>%  
   mutate(Region=gsub("^\\s+|\\s+$", "", Region)) %>% 
   mutate(Missing = ifelse(is.na(Spend), "yes", "no"), Region=tolower(Region), Year=factor(Year), Region=factor(Region))
tidy_ntto_inbound_m <- raw_ntto_inbound_m %>%
   mutate(as.character(TOTAL.ARRIVALS)) %>% 
   gather(key, val, -year, -MONTH) %>%
   mutate(Arrival=as.numeric(gsub('[^0-9]+', '', val)), Month=match(tolower(substr(MONTH, 1, 3)), tolower(month.abb)), key=tolower(gsub('\\.', ' ', key)), Year=year) %>%
   filter(!grepl("total", key)) %>%
   unite(YearMonth, year, Month, sep="-") %>%
   select(Year, YearMonth, MixRegion=key, Inbound=Arrival) %>% 
   mutate(MixRegion=recode(MixRegion, 
                           "latin america  excl mexico "="latin america excl mexico",
                           "prc  excl hk "="china",
                           "roc  taiwan "="taiwan")) %>% 
   mutate(Missing = ifelse(is.na(Inbound), "yes", "no"), Year=factor(Year), Date=as.yearmon(YearMonth), MixRegion=factor(MixRegion)) %>% 
   arrange(MixRegion, Date)
tidy_ntto_outbound_m <- raw_ntto_outbound_m %>% 
   gather(Month, Outbound, -Year, -Regions) %>% 
   mutate(Outbound=as.numeric(gsub('[^0-9]+', '', Outbound)), Month=match(Month, month.abb), MixRegion=tolower(gsub('\\([a-zA-Z0-9]+\\)', '', Regions)), year=Year) %>% 
   mutate(MixRegion=gsub("^\\s+|\\s+$", "", MixRegion)) %>% 
   filter(!grepl("air", MixRegion), !MixRegion=='total overseas', !MixRegion=='north america', !MixRegion=='grand total') %>% 
   unite(YearMonth, year, Month, sep="-") %>% 
   mutate(Region=recode(MixRegion, 
                        "caribbean"="latin america excl mexico",
                        "central america"="latin america excl mexico",
                        "south america"="latin america excl mexico")) %>% 
   mutate(Missing=ifelse(is.na(Outbound), "yes", "no"), Year=factor(Year), Date=as.yearmon(YearMonth), MixRegion=factor(MixRegion), Region=factor(Region)) %>% 
   select(Region, MixRegion, Year, Date, Missing, Outbound) %>% 
   arrange(Region, MixRegion, Date)
tidy_wb_gdp <- raw_wb_gdp %>% 
   select(-Indicator.Name, -Indicator.Code, -X) %>% 
   gather(x_year, GDP, -Country.Name, -Country.Code) %>% 
   filter(grepl(country_str, Country.Name)) %>% 
   separate(x_year, c("X", "Year"), sep="X") %>% 
   select(-X) %>% 
   select(CountryName=Country.Name, CountryCode=Country.Code, Year=Year, GDP=GDP) %>% 
   # filter(year>1999) %>% 
   mutate(Missing = ifelse(is.na(GDP), "yes", "no"))

ggplot(tidy_ntto_spend_y, aes(x=Year, y=Region, fill = Missing)) +
   geom_tile(color = "white") + 
   scale_fill_viridis(discrete = TRUE) +
   ggtitle("Missing Value of NTTO Yearly Spending Data") +
   theme_linedraw()

ggplot(tidy_ntto_inbound_m, aes(x=Date, y=MixRegion, fill = Missing)) +
   geom_tile(color = "white") + 
   scale_fill_viridis(discrete = TRUE) +
   ggtitle("Missing Value of NTTO Inbound Data") +
   theme_linedraw() +
   theme(axis.text.x=element_text(angle=90))

ggplot(tidy_ntto_outbound_m, aes(x=Date, y=MixRegion, fill = Missing)) +
   geom_tile(color = "white") + 
   scale_fill_viridis(discrete = TRUE) +
   ggtitle("Missing Value of NTTO Outbound Data") +
   theme_linedraw() +
   theme(axis.text.x=element_text(angle=90))

ggplot(tidy_wb_gdp, aes(x=Year, y=CountryName,  fill = Missing)) +
   geom_tile(color = "white") + 
   scale_fill_viridis(discrete = TRUE) +
   ggtitle("Missing Value of All Interested Countries (1960-2016)") +
   theme_linedraw() +
   theme(axis.text.x=element_text(angle=90)) 

Executive Summary

Provide a short nontechnical summary of the most revealing findings of your analysis with no more than 3 static graphs or one interactive graph (or link), written for a nontechnical audience. The length should be approximately 2 pages (if we were using pages…) Do not show code, and take extra care to clean up your graphs, ensuring that best practices for presentation are followed.

• Note: the tips below are not intended to be a complete list of everything we’ve covered this semester on designing a successful graph. It’s meant to help you avoid some common problems.

• Title, axis labels, tick mark labels, and legends should be comprehensible (easy to understand) and legible (easy to read / decipher).

• Tick marks should not be labeled in scientific notation or with long strings of zeros, such as 3000000000. Instead, convert to smaller numbers and change the units: 3000000000 becomes “3” and the axis label “billions of views”.

• Units should be intuitive (Extreme example: an axis labeled in month/day/year format is intuitive, one labeled in seconds since January 1, 1970 is not.)

• The font size should be large enough to read clearly. The default in ggplot2 is generally too small. You can easily change it by passing the base font size to the theme, such as + theme_grey(16). (The default base font size is 11.)

• The order of items on the axes and legends is logical. (Alphabetical is often not logical.)

• Colors should be color vision deficiency friendly.

• If a legend is taking up too much space on the right, move it to the bottom.

• If categorical variable levels are long, set up the graph so the categorical variable is on the y-axis and the names are horizontal. A better option, if possible, is to shorten the names of the levels.

• Not all EDA graphs lend themselves to presentation, either because the graph form is hard to understand without practice or it’s not well labeled. The labeling problem can be solved by adding text in an image editor. The downside is that it is not reproducible. If you want to go this route, Paintbrush is a free and simple bitmap image editor for the Mac: https://paintbrush.sourceforge.io/ There are many other options.

• Err on the side of simplicity. Don’t, for example, overuse color when it’s not necessary. Ask yourself: does color make this graph any clearer? If it doesn’t, leave it out.

• Test your graphs on nontechnical friends and family and ask for feedback.

Main Analysis

Provide a detailed, well-organized description of your findings, including textual description, graphs, and code. Your focus should be on both the results and the process. Include, as reasonable and relevant, approaches that didn’t work, challenges, the data cleaning process, etc.

• The guidelines for the Executive Summary above do NOT apply to exploratory data analysis. Your main concern is designing graphs that reveal patterns and trends.

• As noted in Hmk #4, do not use circles, that is: bubbles, pie charts, or polar coordinates.

• Use stacked bar charts sparingly. Try grouped bar charts and faceting as alternatives, and only choose stacked bar charts if they truly do a better job than the alternatives for observing patterns.

Data Cleaning

Since the data is very messy, we put many effort on cleaning and extract useful infomation for analysis.

1. Convert to correct type
2. Consolidate name, region, date

Join same region

region_str <- "africa|asia|canada|latin america (excl mexico)|europe|mexico|middle east|oceania"
inbound_region <- tidy_ntto_inbound_m %>% 
   filter(grepl(region_str, MixRegion)) %>% 
   select(Region=MixRegion, Year, Date, Inbound) %>% 
   group_by(Region, Year, Date) %>% 
   summarise(TotalInbound=sum(Inbound)) %>% 
   ungroup
outbound_region <- tidy_ntto_outbound_m %>% 
   select(Region, Year, Date, Outbound) %>% 
   group_by(Region, Year, Date) %>% 
   summarise(TotalOutbound=sum(Outbound)) %>% 
   ungroup
regional_travel <- inner_join(inbound_region, outbound_region, 
                              by=c("Region"="Region", "Year"="Year", "Date"="Date"))

Challenges

There are several challenges in our project:

1. Due to the problems in the data set such as inconsistency, we have to spend much time in cleaning and re-organizing it, which makes the work tedious and laborious.
2. We need country level data in some graphs; however, what we can acquire from the dataset is region level. In that case, we have to project the data onto the whole region, which makes the analysis not comprehensive and detailed.
3. Shiny is a great tool for creating interactive data visualizations in R; however, we do not have much experience in it, and therefore have to spend time learning it, which is not easy in such a short time.

Analysis

Travel and Tourism Analysis

regional_travel %>% 
   select(Year, TotalInbound, TotalOutbound) %>% 
   group_by(Year) %>% 
   summarise(TotalInbound=sum(TotalInbound), TotalOutbound=sum(TotalOutbound)) %>% 
   plot_ly(x = ~Year, y = ~TotalInbound, type = 'bar', name = 'Inbound', marker = list(color = 'rgb(55, 83, 109)')) %>%
add_trace(y = ~TotalOutbound, name = 'Outbound', marker = list(color = 'rgb(26, 118, 255)')) %>%
layout(title = 'Yearly Inbound and Outbound',
      xaxis = list(
        title = "",
        tickfont = list(
          size = 14,
          color = 'rgb(107, 107, 107)')),
      yaxis = list(
        title = 'Number of People',
        titlefont = list(
          size = 16,
          color = 'rgb(107, 107, 107)'),
        tickfont = list(
          size = 14,
          color = 'rgb(107, 107, 107)')),
      legend = list(orientation = 'h', x = 0, y = 1, 
                    bgcolor = 'rgba(255, 255, 255, 0)', bordercolor = 'rgba(255, 255, 255, 0)'),
      barmode = 'group', bargap = 0.15)

The United States is one of the largest destinations for visitors and has a large amount outbound journeys as well. The numbers of inbound and outbound have increased a lot from 2009 to 2010. Between 2010 to 2013, the number of international visitors raised a little bit each year however the amound of outbound is kind of stable. Since 2013, both of them grown a lot and finally achieved 77.5 million international visitations and 85.6 million outbound travellers in 2015.

Naturally we start wondering what are the most popular destination for americans and where are these international vistors come from? To answer these questions, we break down the graph into smaller regions.

p1 <- inbound_region %>% 
   spread(Region, TotalInbound) %>% 
   filter(Date>'2008-11') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~africa, name='africa', mode='lines', line = list(color="gray", width = 1)) %>%
   add_trace(y=~asia, name='asia', mode='lines', line = list(color="red", width = 1)) %>%
   add_trace(y=~canada, name='canada', mode='lines', line = list(color="orange", width = 1)) %>%
   add_trace(y=~europe, name='europe', mode='lines', line = list(color="pink", width = 1)) %>%
   add_trace(y=~`latin america excl mexico`, name='latin america excl mexico', mode='lines', line = list(color="green", width = 1)) %>%
   add_trace(y=~mexico, name='mexico', mode='lines', line = list(color="purple", width = 1)) %>%
   add_trace(y=~`middle east`, name='middle east', mode='lines', line = list(color="black", width = 1)) %>%
   add_trace(y=~oceania, name='oceania', mode='lines', line = list(color="blue", width = 1)) 
p2 <- outbound_region %>% spread(Region, TotalOutbound) %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~africa, name='africa', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~asia, name='asia', mode='lines', line = list(color="red", width = 1), showlegend=F) %>%
   add_trace(y=~canada, name='canada', mode='lines', line = list(color="orange", width = 1), showlegend=F) %>%
   add_trace(y=~europe, name='europe', mode='lines', line = list(color="pink", width = 1), showlegend=F) %>%
   add_trace(y=~`latin america excl mexico`, name='latin america excl mexico', mode='lines', line = list(color="green", width = 1), showlegend=F) %>%
   add_trace(y=~mexico, name='mexico', mode='lines', line = list(color="purple", width = 1), showlegend=F) %>%
   add_trace(y=~`middle east`, name='middle east', mode='lines', line = list(color="black", width = 1), showlegend=F) %>%
   add_trace(y=~oceania, name='oceania', mode='lines', line = list(color="blue", width = 1), showlegend=F)
subplot(p1, p2, nrows=2, shareX=T) %>%
  layout(title = "Inbound v.s. Outbound",
         yaxis = list(title = "Inbound"),
         yaxis2 = list(title = "Outbound"),
         legend = list(orientation = 'h')
         )

After separating each region out, we observed several things:

• Most of the international travellers are come from Canada, Mexico, Europe, and Asia.
• Mexico, Canada, Europe, and Latin America except Mexico are the top destinations for americans.
• Seasonality exists in each line. Usually peak is reached in summer. For example, every year of July, the number of canada vistors reaches its highest peak of the year.
• A boom in amount of visitors from Latin America except Mexico in the begining of 2014 and a boom in amount of people travel to Mexico in the start of 2010.

We further draws inbound and outbound per region to better explore the hidden pattern individually.

p1 <- regional_travel %>% 
   filter(Region=='africa') %>% 
   plot_ly(x = ~as.POSIXct(Date), height = 1000) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1)) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1)) %>% 
   layout(autosize=F)
p2 <- regional_travel %>% 
   filter(Region=='asia') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p3 <- regional_travel %>% 
   filter(Region=='canada') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p4 <- regional_travel %>% 
   filter(Region=='europe') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p5 <- regional_travel %>% 
   filter(Region=='latin america excl mexico') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p6 <- regional_travel %>% 
   filter(Region=='mexico') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p7 <- regional_travel %>% 
   filter(Region=='middle east') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p8 <- regional_travel %>% 
   filter(Region=='oceania') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
subplot(p1, p2, p3, p4, p5, p6, p7, p8, nrows=8) %>%
  layout(title = "Regional Inbound and Outbound",
         yaxis = list(title = "Africa"),
         yaxis2 = list(title = "Asia"), 
         yaxis3 = list(title = "Canada"),
         yaxis4 = list(title = "Europe"),
         yaxis5 = list(title = "Latin America"),
         yaxis6 = list(title = "Mexico"),
         yaxis7 = list(title = "Middle East"),
         yaxis8 = list(title = "Oceania"),
         legend = list(orientation = 'h', x = 0, y = 1.005)
         )

• Africa: There are more and more people from Africa travel to U.S. since 2013, but the number of american who go to Africa is very stable for past 7 years.

• Asia: More and more Asias come to U.S, however, less Americans go to Asia after May 2010. Another thing to notice is the number of visitors from Asia grows faster in July compare to other month. This is probabaly due to the increase of foreign students.

• Canada: Not many people go to Canada during year 2009. It may infuluenced by the Global financial crisis in 2009.

• Europe: Different than other regions, the trend of inbound and outbound is very close to each other, which means that the amount of people leave

• Latin America except Mexico: A huge increase occured in 2014. Since it boosted too much, we question the correctness of the data source.

• Mexico: A boom of number of outbound happens in the end of 2009. We did a lot research online and found many news with topics related to More Mexicans Leaving Than Coming to the U.S during that time. We suspicious this could be the reason. Unlike other regions, there’s no obvious peak for Mexico.

• Middle East: Both the number of inbound and outbound increase by year.

• Oceania: Like Asia, less of U.S. citizens visit Oceania region since mid 2010.

Spend Analysis

yearly_spend <- tidy_ntto_spend_y %>% 
   filter(Region!='european union', Region!='south-central america', Region!='overseas') %>% 
   mutate(Region=recode(Region, "asia-pacific"="asia"), Spend=Spend*1000000) %>% 
   select(-Missing) %>% 
   arrange(Region, Year, Type, Category) 

yearly_spend %>%
   group_by(Type, Year) %>%
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Type, TotalSpend) %>%
   ungroup %>% 
   plot_ly(x = ~Year) %>%
   add_trace(y=~`Payments (imports)`, type="scatter", name='Payments (imports)', mode = 'lines+markers', line = list(color="blue", width = 2)) %>%
   add_trace(y=~`Receipts (exports)`, type="scatter", name='Receipts (exports)', mode = 'lines+markers', line = list(width = 2)) %>% 
   layout(title = "Yearly Spending", 
          xaxis = list(title = "Year"), yaxis = list(title = "Spend"), 
          legend = list(orientation = 'h', x = 0.5, y = 1.005))

From the graph we can see that more money has been spent by international vistors in U.S. than americans travel outside. Spend increases by year overall, however, there’s a big drop in 2009 when the Global Financial Crisis happens.

p1 <- yearly_spend %>% 
   filter(Type=="Payments (imports)") %>% 
   select(-Type) %>% 
   group_by(Region, Year) %>% 
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Region, TotalSpend) %>% 
   plot_ly(x = ~Year) %>%
   add_trace(y=~africa, type="scatter", name='africa', mode = 'lines+markers', line = list(color="gray", width = 2)) %>%
   add_trace(y=~asia, type="scatter", name='asia', mode = 'lines+markers', line = list(color="red", width = 2)) %>%
   add_trace(y=~europe, type="scatter", name='europe', mode = 'lines+markers', line = list(color="blue", width = 2)) %>%
   add_trace(y=~`latin america`, type="scatter", name='latin america', mode = 'lines+markers', line = list(color="green", width = 2)) %>%
   add_trace(y=~`middle east`, type="scatter", name='middle east', mode = 'lines+markers', line = list(color="orange", width = 2))
p2 <- yearly_spend %>% 
   filter(Type=="Receipts (exports)") %>% 
   select(-Type) %>% 
   group_by(Region, Year) %>% 
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Region, TotalSpend) %>% 
   plot_ly(x = ~Year) %>%
   add_trace(y=~africa, type="scatter", name='africa', mode = 'lines+markers', line = list(color="gray", width = 2), showlegend=F) %>%
   add_trace(y=~asia, type="scatter", name='asia', mode = 'lines+markers', line = list(color="red", width = 2), showlegend=F) %>%
   add_trace(y=~europe, type="scatter", name='europe', mode = 'lines+markers', line = list(color="blue", width = 2), showlegend=F) %>%
   add_trace(y=~`latin america`, type="scatter", name='latin america', mode = 'lines+markers', line = list(color="green", width = 2), showlegend=F) %>%
   add_trace(y=~`middle east`, type="scatter", name='middle east', mode = 'lines+markers', line = list(color="orange", width = 2), showlegend=F)
subplot(p1, p2, shareY=T) %>%
  layout(title = "Payments vs Receipts by Region",
         yaxis = list(title = "Spend"),
         legend = list(orientation = 'h', y=-0.15)
         )

In fact, payments is quiet stable by year, these’s no big jump or boom, however receipts is a different story. After breaking down by region, it is clear that, the total amount and increasing rate of Asia region is much more than others. And the Global Financial Crisis has a big influence on the money spent by visitors from Latin America.

p1 <- yearly_spend %>% 
   filter(Type=="Payments (imports)") %>% 
   select(-Type, -Region) %>% 
   group_by(Category, Year) %>% 
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Category, TotalSpend) %>% 
   ungroup %>% 
   plot_ly(x = ~Year) %>%
   add_trace(y = ~Education, name = 'Education', type = 'scatter', mode = 'lines+markers', line = list(color="blue")) %>% 
   add_trace(y = ~`Medical/Short-Term Workers`, name = 'Medical/Short-Term Workers', type = 'scatter', mode = 'lines+markers', line = list(color="orange")) %>% 
   add_trace(y = ~`Other Business/Other Personal Travel`, name = 'Other Business/Other Personal Travel', type = 'scatter', mode = 'lines+markers', line = list(color="green"))
p2 <- yearly_spend %>% 
   filter(Type=="Receipts (exports)") %>% 
   select(-Type, -Region) %>% 
   group_by(Category, Year) %>% 
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Category, TotalSpend) %>% 
   ungroup %>% 
   plot_ly(x = ~Year) %>%
   add_trace(y = ~Education, name = 'Education', type = 'scatter', mode = 'lines+markers', line = list(color="blue"), showlegend=F) %>% 
   add_trace(y = ~`Medical/Short-Term Workers`, name = 'Medical/Short-Term Workers', type = 'scatter', mode = 'lines+markers', line = list(color="orange"), showlegend=F) %>% 
   add_trace(y = ~`Other Business/Other Personal Travel`, name = 'Other Business/Other Personal Travel', type = 'scatter', mode = 'lines+markers', line = list(color="green"), showlegend=F) 
subplot(p1, p2, shareY=T) %>%
  layout(title = "Payments vs Receipts by Category",
         yaxis = list(title = "Spend"),
         legend = list(orientation = 'h', y=-0.15)
         )

Finnaly, we look at three categories of spend. Other Business/Other Personal Travel has the biggest amount in both payments and receipts. It would be nice if this category can be break down to smaller sub groups, but unfortunately we do not have access to more detailed data. One thing is worth to mention is, unlike other two categories, the Global Financial Crisis did not influence Education.

Conclusion

Discuss limitations and future directions, lessons learned.

A note on style: You are encouraged to be as intellectually honest as possible. That means pointing out flaws in your work, detailing obstacles, disagreements, decision points, etc. – the kinds of “behind-the-scene” things that are important but often left out of reports. You may use the first person (“I”/“We”) or specific team members’ names, as relevant.

The goal of this project is to answer the four questions raised above. We have accomplished the task despite of some flaws and limitations. For instance, the latest data we could find is for the year 2015, which is more than one year ago. In that case, we could not say the analysis is very persuasive since the data source is not up to date. We are unable to get data that are more detailed so that we cannot dig deeper to find root causes of some observations. At the very beginning, we thought the data was exhaustive and comprehensive and we made plenty of hypotheses and expectations based on the first impression; however, we find that most of our ideas are not realistic as we investigate the data more. Unfortunately, we have to discard some interesting ideas due the inconstancy in the dataset.

For the future work, we could introduce more supportive data such as the inbound and outbound traveler data of other countries, and then compare it with the USA. We believe that it would reveal results that are more interesting.